Two-level Synchronization using Particle Filter for Co-player Music Robots

Our goal is to develop a co-player music robot, i.e., a robot that presents a musical expression together with humans. A music interaction requires two important functions: synchronization with the music and musical expression, such as dancing or playing a musical instrument. Many instrumentperforming robots are only capable of the latter function, they may have difficulty in playing live with human performers. The synchronization function is critical for the interaction. We classify synchronization and musical expression into two levels: (1) the rhythm level and (2) the melody level. Two issues in achieving two-level synchronization and musical expression are: (1) simultaneous estimation of the rhythm structure and the current part of the music and (2) derivation of the estimation confidence to switch behavior between the rhythm level and the melody level. This paper presents a score following algorithm, incremental audio to score alignment, that conforms to the twolevel synchronization design using a particle filter. Our method estimates the score position for the melody level and the tempo for the rhythm level. The reliability of the score position estimation is extracted from the probability distribution of the score position. Experiments are carried out using 20 polyphonic jazz songs. The results confirm that our method switches levels to alleviate the error in the score position estimation. When some of the musical notes sound vague or when temporal fluctuations are frequently observed, the estimated score position tends to be erroneous. In the experiment, these errors turn out to be reduced by our two-level synchronization strategy.